Document

Contents lists available at ScienceDirect

Journal of Sea Research

journal homepage: www.elsevier.com/locate/seares

Time series prediction of sea surface temperature based on BiLSTM model with attention mechanism

Nabila Zirra $^{a}$, Assia Kamal-Idrissi, Rahma Farsi $^{c}$, Haris Ahmad Khan d, e,

$^{a}$National Superior School of Mines Rabat, LISTD Laboratory, ADOS Team, Rabat, Morocco

$^{b}$Mohammed VI Polytechnic University, Ai movement, Center of Artificial Intelligence, Rabat, Morocco
$^{c}$National Superior School of Mines Rabat, Morocco
$^{d}$Agricultural Biosystems Engineering, Wageningen University & Research, Wageningen, the Netherlands
$^{e}$Data Science, Crop Protection Development, Syntenga, the Netherlands

A R T I C L E I N F O

Keywords:

Bidirectional Long Short-Term Memory

(BILSTM)

Attention

Sea Surface Temperature (SST)

Prediction

Marine data

Morocco

1. Introduction

Over the past two decades, satellite instruments and in situ observations, including marine stations, buoys, and voluntary observing ships, have provided valuable measurements of the physical characteristics of the ocean surface. These observations, despite their varying spatial and temporal sampling and measurement techniques, have become indispensable for understanding ocean dynamics. Notably, the use of altimetry data to study dynamic ocean topography has significantly advanced our comprehension of medium-to-large ocean variability (over 100-200 km) (Wang et al., 2022).

Sea Surface Temperature (SST) is a complex ocean parameter with significant implications for climate and marine ecosystems. Analyzing SST enables the management of marine ecosystems, as it provides insights into ocean conditions and climatic dynamics (Xuan et al., 2020). Accurate prediction of SST is of utmost importance, ranging from short-

* Corresponding author at: Agricultural Biosystems Engineering, Wageningen University & Research, Wageningen, the Netherlands. E-mail address: haris.khan@wur.nl (H.A. Khan).

https://doi.org/10.1016/j.seares.2024.102472

Received 5 October 2023; Received in revised form 2 January 2024; Accepted 18 January 2024

Available online 24 January 2024

term to long-term forecasts. These predictions aid decision-making processes related to tasks such as fisheries resource distribution and marine environmental protection. Consequently, SST prediction plays a critical role in various domains, including marine fisheries, weather forecasting, marine animal tracking, and ship path planning.

Recent literature has witnessed an active research focus on SST prediction. This field involves generating future SST values in advance using predictive models with different time horizons (e.g., minutes, hours). However, SST data is often incomplete and structured as a time series, featuring complex or irrelevant coupling mechanisms and spatiotemporal relationships. These characteristics pose challenges to traditional data analysis approaches. To overcome these limitations, researchers have developed Machine Learning (ML) and Deep Learning (DL) models, which have demonstrated greater robustness and accuracy. DL methods, such as Recurrent Neural Networks (RNNs), have been employed in previous studies to predict SST time series. Long Short-

Term Memory (LSTM), a type of RNN, is particularly popular due to its ability to retain information over long periods of time. Numerous variations of LSTM have been proposed to enhance prediction performance by leveraging historical data to predict SST (Kun et al., 2021; Fei et al., 2022). The attention mechanism assigns weights to historical data, enhancing the utilization of historical temporal information. However, these versions of LSTM process sequences in one direction, limiting their ability to utilize future information.

To address this limitation, a Bidirectional Long Short-Term Memory neural network (BilSTM) was developed, which consists of two LSTM models operating in both backward (future to past) and forward (past to future) directions. This architecture combines the advantages of sequential data processing and the long-term memory capabilities of both forward and backward LSTMs (Nie et al., 2021).

This paper proposes an attention-bilSTM model that effectively captures the relationships between historical and future SST values by employing a robust attention mechanism. The combination of attention and BilSTM has demonstrated its effectiveness in similar time series prediction tasks in some previous works (Ahlam et al., 2022; Lee et al., 2022; Yang and Wang, 2022; Ma et al., 2020). Previous studies on

marine data in Morocco have primarily focused on forecasting seasonal precipitation (Tuel and Elatahir, 2018). To the best of our knowledge, this is the first attempt to utilize the BilSTM with an attention mechanism model for SST prediction in marine data, specifically in the context of Morocco. The main contributions of this paper are:

· The use of BilSTM with optimal hyperparameters using K-fold crossvalidation captures good temporal patterns in SST time series forecasting.
· The use of the attention mechanism allows BilSTM to focus on relevant temporal patterns and enhances its ability to capture completeness relationships within time series data. Potentially, it improves forecasting accuracy and provides valuable interpretability for SST time series data.
· A series of experiments in four Moroccan cities were conducted to evaluate the proposed model. Also, the attention-BilSTM model was compared to other recurrent neural networks, traditional machine learning models, a gradient boosting model, and the recent deep learning model. In all experiments, the proposed model showed the highest prediction results.

The remainder of this paper is organized as follows. Section 2 provides a review of relevant literature on the time series prediction of SST. Section 3 introduces the proposed BILSTM with an attention method for SST prediction. Section 4 describes the study area and presents the experimental results. Finally, Section 5 concludes the paper.

2. Related work

The prediction methods of SST can be roughly divided into two types: numerical methods and data-driven methods, where data-driven models can be further categorized into statistical models, machine learning models, and deep learning models. Data-driven methods encompass various statistical and artificial intelligence techniques, including ML methods (Ali et al., 2021). In contrast to numerical models, data-driven approaches require more data but less prior knowledge, resulting in improved prediction accuracy. Figure 1 presents the taxonomy of SST prediction models.

2.1. Numerical methods

Real-time forecasts of SST are provided by many agencies around the world, namely the European Centre for Medium-Range Weather Forecasts (ECMWF) and the Command National Centers for Environmental Prediction (NCEP). These methods of SST prediction consist of modeling the physical environment parameters based on kinetic and thermodynamic equations (Xie et al., 2019). They are widely used in large seas. Many authors have used coupled Generalulated Models (GCMs) to predict SST over a basin. Krishnamurath et al. (2006) used thirteen state-of-the-art coupled global atmosphere-ocean models to predict seasonal global SST anomalies.

Besides, the Integrated Forecast System (IFS) is used by ECMWF and the Global Forecast System (GFS) by NCEP, respectively. The IFS forecasts the SST of the following 10-15 days while the NCEP forecasts the SST of the following 380 h (Aparna et al., 2018).

2.2. Data-driven methods

Data-driven prediction methods use statistical methods to predict SST. This kind of method requires a large amount of data but less knowledge of the oceans and atmosphere due to its capability to learn patterns from historical data automatically and further use the learned patterns to predict future values of SST. This category of methods includes artificial intelligence, namely machine learning and deep

learning models. They outperform highly complex computations with excellent time series prediction results (Shi et al., 2022).

2.2.1. Statistical methods

Statistical methods learn from historical data to predict SST. They range from simple regression, such as linear regression, to complex methods like the Markov model (Xue and Leetmaa, 2000). Autoregressive Mixed Growing Average (ARIMA), which is a Regression model, has been widely used for SST predictions due to its simplicity, adaptability, and the Box-Jenkins methodology (de Matos et al., 2022). The latter is used to provide a well-established design process for linear patterns.

2.2.2. Machine Learning methods

With the continuous development of technology, ML models have been efficiently used as an alternative to statistical methods for different tasks due to their nonlinearity, performance, and flexibility. They have shown an important capacity and great potential to predict the parameters of the ocean and weather. Among these models are Support Vector Machines (SVM) and XGBoost. The latter is an option to describe SST's characteristics, which are complex temporal-dependence structure and multi-level seasonality (Wolf et al., 2020). Meanwhile, SVM constructs a set in a high-dimensional space that can be used in regression task (Hou et al., 2022). For example, Lins et al. (2013) proposed an SVM combined with multi-linear regression (MR) to predict SST over the Northeastern Brazilian Coast and the tropical Atlantic.

Besides, the Support Vector Regression (SVR) model achieves a satisfactory result in predicting SST. Basically, these ML models outperform the numerical models in terms of accuracy and simplicity (Lee et al., 2016; Rehana, 2019). However, these kinds of models are not able to properly model nonlinear patterns in temporal phenomena.

2.2.3. Deep Learning methods

As ML models, Deep Learning models have recently experienced rapid development due to their ability to model complex relationships by extracting features from hidden data. They are widely used in many applications, such as time series prediction tasks. Choi et al. (2022) proposed a new hybrid method to enhance the accuracy of a real-time ocean forecasting system by combining a deep Generative Inapainting Network (GIN) and a numerical ocean model. The most exciting studies specifically focused on temperature-related tasks using CNNs have been those that employed satellite data. Since the prediction of SST is basically a time series problem, the use of the LSTM architecture is common

for this task. Xiao et al. (2019) used a combination of CNN and LSTM, which is a spatio-temporal deep learning model, to forecast SST. The proposed model used satellite data, rain gauge data, and thermal infrared images. To extract spatial features, the CNN model was used for this objective, meanwhile, the LSTM model aims to handle the time dependencies of the provided data. Results show that the CNN-LSTM model outperforms comparative models, such as CNN, LSTM, and MLP. They had relatively accurate predictions results for the short-term and mid-daily forecast for SST (SIT et al., 2020). The MLP was exploited by Wolff et al. (2020) to predict the short and long-term evolution of the SST. The algorithm was able to forecast SST with hindcast input and atmospheric data comparable to a state-of-the-art physics-based model simulation from the European Center for Medium Weather Forecasting.

Zheng et al. (2020) proposed a model combining Deep Neural Network (DNN) and bias correction maps. DNN is composed of four stacked composite layers. Each layer processes SST maps at different resolutions to analyze systems at multiple scales simultaneously. The forecast results were validated by satellite observations in the eastern equatorial Pacific Ocean. Zhang et al. (2021) designed a new Memory Graph Convolutional Network (MGCN) that captures spatio-temporal changes of SST. MGCN consists of two: the memory layer and the graph layer. The first layer encodes temporal changes using gate linear units and convolution units. Whereas, the second layer captures spatial changes in the frequency domain using Laplacian. More recently, Qiao et al. Qiao et al. (2023) suggested an ensemble learning method that combines a spatio-temporal deep learning network with the attention mechanism. The authors used the XGBoost algorithm to extract seasonal periodic features from SST data. Then, they extracted spatio-temporal features using PredRND

(Predictive Recurrent Neural Network). Finally, they added an attention mechanism to extract the important patterns in historical SST data and further improve the prediction accuracy. All these studies have shown that deep learning techniques have the advantages of strong nonlinear mapping and multidimensional information processing.

3. Methodology

The suggested pipeline for predicting future SST values is shown in Fig. 2. The approach consists of two major phases: data preparation and network training. First, we select the study region in which the SST values will be predicted. The data represents a time series that may encounter several problems, such as missing values (or timestamps), and outliers. For that, a processing step is required to handle these problems. Moreover, we normalize time series data to improve the training stability as well as the performance of the Attention-BiLSTM network. Then, the processed time series data is split into training and testing sets. Second, we establish the Attention-BiLSTM model to perform accurate time series prediction. Finally, the best model with the optimized hyperparameters is saved for further use.

3.1. Study area and data

Morocco is the northwesternmost country in Africa. Its sea surface spans the Mediterranean Sea and the Atlantic Ocean on the north and west, respectively. In this work, SST value prediction was carried out in four different Moroccan cities: Tangier, Agadir, Casablanca, and Kariat Arkmane. As shown in Fig. 3, the choice of these cities is based on their location relative to the sea. Agadir is located on the shore of the Atlantic Ocean near the foot of the Atlas Mountains. Casablanca is the

Region	Coordinates
Tangier	36.250261525348487, -5.9773927942819712
Agadir	31.1460875001933, -9.66901533325119
Casablanca	33.9772355675074, -7.849345740762594
Kariat Arkmanne	35.12039529923372, -2.73743916487201

Region	Mean	Warmest	Coldest	STD
Tangier	18.360108	24.19	14.62	2.266212
Azadir	18.349655	22.77	1.54	1.692016
Casablanca	19.106891	24.17	14.79	2.373244
Kariat Arkmanne	19.549607	27.63	13.76	3.709748

largest city in Morocco. It is located on the Atlantic coast of the Chaoua Plain, in the central-western part of Morocco. Kariat Arkmanne is a coastal town in Morocco, in the Oriental region. It is located in the Nador province, 25 km from the city of Nador. Kariat Arkmanne is served by the Mediterranean Ring Road, which crosses it. Tangier is a city in northwestern Morocco. The Atlantic Ocean meets the Mediterranean Sea on the coast at the western entrance to the Strait of Gibraltar. Table 1 depicts the GPS coordinates of each selected city in the study area.

The measurements of the SST are provided by the daily satellite readings provided by the National Oceanic and Atmospheric Administration (NOAA) from January 01, 2012, to August 24, 2022. Table 2 summarizes statistics related to SST data, including measures of the mean, standard deviation, and specific extremes value such as minimum and maximum temperature. The choice of these four cities also depends on statistics, which are similar in most cities. In deep learning, using data with similar statistics can help in building models that generalize well. Models tend to perform better when applied to new, unseen data that follows the same distribution. Moreover, similar statistics can be beneficial when splitting data into training and testing sets. It ensures that both sets have comparable statistical properties, which helps in creating reliable models and assessing their performance accurately.

3.2. Data preprocessing

Predictive models for time series require clean and complete data without outliers or missing values. Therefore, it is crucial to apply appropriate preprocessing techniques before utilizing these. In this study, we employed standard preprocessing methods to enhance the quality of the data, including conventional inputor handling missing values.

Furthermore, it is essential to normalize the time series data due to its varying measurement scales. Deep learning methods aiming to map input data to output across all samples in the training set. The model initializes the weights randomly and updates them using optimization techniques like Stochastic Gradient Descent (SGD) or Root Mean Square Propagation (RMSRPop). The initial weights and the error calculated between the predicted and actual values emphasize the significance of scaling both the input and output. Failure to scale the time series data appropriately in prediction problems can lead to an unstable learning process.

To handle this problem, we perform a min-max normalization technique to transform time series data in the range (0.1). Equation1 depicts the feature scaling formula:

\chi _ { x c a l e d } = \frac { x - x _ { \min } } { x _ { \max } - x _ { \min } }

Where x represents the origin value, x$_{min}$ and x$_{max}$ are the lowest and highest values, respectively.

The time series data used in this study spans a period of 11 years, specifically from January 1, 2012, to August 24, 2022, encompassing a total of 3889 days. To divide the dataset into training and testing sets, the data from January 1, 2012, to December 31, 2021 (a total of 3653 days) is allocated for training purposes. The remaining data, consisting of 236 days, is reserved for testing the model's performance.

For a visual representation of the data distribution across the study area, please refer to Fig. 4. This figure illustrates the training and testing data for each region within the study area.

3.4. Bidirectional Long Short-Term Memory (BILSTM)

Long Short-Term Memory (LSTM) is a specialized type of Recurrent Neural Network (RNN) that incorporates Long Short-Term Memory units. These units were introduced by Hochreiter and Schmidhuber, German researchers, as a solution to the vanishing gradient problem (Hochreiter and Schmidhuber, 1997). LSTMs have proven to be highly effective across a wide range of problem domains and are now widely utilized in various applications.

LSTMs consist of memory blocks, which contain memory cells with self-connections. These cells retain the temporal state of the network. Additionally, LSTMs incorporate multiplicative units called gates, which enable the storage, retrieval, and modification of information within the cells. Each cell autonomously determines which information to store and controls the opening and closing of gates for reading, writing, and resetting. The gates operate in a binary manner, with two states: open and closed. However, these gates are implemented using element-wise multiplication by the sigmoid function, which restricts their values to open range of 0 to 1. This ablation implementation enables differentiability and facilitates the backpropagation algorithm. The gates process the signals they receive, selectively allowing or blocking information based on its relevance and importance, which is determined by their respective weights. These weights are adjusted during the recurrent network learning process, modulating the input and hidden states of the LSTM. First, the LSTM unit relies on deciding what information seems important to keep. The decision is made by the forget layer, which looks at x$_{i}$ and h$_{i}$ - 1 , then outputs a value between 0 and 1 in the cell state c$_{t}$$_{-}$$_{1}$ using a sigmoid function.

f _ { i } = \omega ( x _ { i } y _ { i } + u _ { i } h _ { i } + u _ { i } )

Second, the LSTM unit selects the new information that should be stored in the cell state. The input gate layer i decides which values will be updated using the sigmoid function. Then, a tanh function creates a vector of new candidate values ' that could be added to the state.

i = \omega _ { i } ( x _ { i } u _ { i } + u _ { i } h _ { i } + b )

Third, the old state c$_{t}$$_{-}$$_{1}$ is updated into the new cell state c$_{t}$ .

c _ { t } = f _ { i } c _ { t - 1 } + i + \iota _ { i } c _ { t }

Finally, the output gate o , decides the amount of memory content to yield to the next hidden state.

o _ { i } = \omega ( x _ { i } w _ { i } + u _ { i } h _ { i } + b )

h _ { i } = o _ { i } - \tanh ( c _ { i } )

Where ( · ) is the inner products, w$_{0}$ , and u$_{0}$ are the weights, and b$_{0}$ is the bias.

As shown in Fig. 5, BILSTM represents an amelioration of the LSTM model that contains two separate LSTM networks: backward and forward (Graves and Schmidhuber, 2005; Graves et al., 2005). The forward hidden state is calculated by Eq. (8):

$_{¯}$ h = o$_{·}$ -tanh ( c$_{i}$ )

Where: o$_{i}$ represents the output gate and c$_{i}$ represents the cell state. Similarly, the backward hidden state ¯ h is calculated to the forward

Explanation of BIBLSTM parameters.
Parameter	Explanation
x	The input vector at time t (i.e., SST time series)
f	The output of the forget gate
i	The output of the input gate
c$_{i}$	The vector of candidate values that should be added to the cell state
c$_{i}$	The new cell state
h$_{i}$	The output of the output gate
h$_{i}$	The hidden state at time t
y$_{i}$	The final predicted output at time t

Finally, an activation function is applied to the hidden state h$_{i}$ in order to generate the final output y$_{i}$ . Table 3 gives more explanation about all parameters of the BIBLSTM equations.

3.5. Attention mechanism

The attention mechanism of humans leads us to selectively focus on the important information required and ignore other visible information that seems irrelevant (Niu et al., 2021). Nowadays, attention is extensively applied to computer vision (Wang and Tax, 2016), natural language processing (Galaissi et al., 2020), machine translation (Bahdanau et al., 2014; Luong et al., 2015), time series prediction (Noor et al., 2022), and other fields (Yang et al., 2016; Wang et al., 2016).

The attention mechanism helps models to provide an ability to focus only on crucial features from all the data. Deep Learning adopted the attention mechanism in time series models to improve the accuracy of predicting.

The proposed approach adds an attention layer after three BIBLSTM layers. The output of the last BIBLSTM layer represents the input of the attention layer that will assign more weights to SST features that seem important in predicting. At this stage, the approach computes the probability according to the weight distribution. Then, it updates and optimizes weight parameters at each iteration during training. The attention mechanism can be expressed by the following formulas (10,11,12):

A _ { i } = \ v \, \tanh ( \omega _ { h } + b ) \\ \alpha _ { i } = \exp ( A _ { i } ) \\ \frac { \sum _ { i } A _ { i } } { \sqrt { 1 } }

bidirectional_input input: [(3593, 60, 1)] [(3593, 60, 1)] InputLayer output:

	bidirectional(lstm) input: (3593, 60, 1) (3593, 60, 256)
bidirectional(LSTM) output:

	bidirectional_1(lstm_1) input: (3593, 60, 256) (3593, 60, 256)
bidirectional(LSTM) output:

	dropout 1 input: (3593, 60, 256) (3593, 60, 256)
Dropout output:

	attention input: (3593, 60, 128) (3593, 128)
Attention output:

Fig. 6. Attention-BIBLSTM architecture.

O _ { t } = \sum _ { t = 1 } ^ { i } a _ { t, h _ { t } }

Where:

· A$_{i}$ : represents the attention probability distribution;
· v and w : represent the weight coefficients;
· b : represents the bias coefficient;

Zrina et al. Journal of Sea Research 198 (2024) 102472

· O$_{i}$ represents the attention layer output at time t .

To select the number of layers and neurons in our model, we performed a k-fold cross-validation (5 fold), which plays a crucial role in assessing model performance, tuning hyperparameters, and evaluating generalization ability. This technique involves systematically exploring various architectures and hyperparameters to find the best combination. The best architecture of the Attention-BiLSTM model is depicted in Fig. 6, which comprises several layers. It begins with an input layer, followed by three bidirectional LSTM layers, two dropout layers, an attention layer, and a dense layer. The input layer has a shape of (3593, 60, 1), where 3593 represents the training data and 60 represents the number of time steps. Determining the optimal number of time steps in neural networks, particularly for LSTM models, is not straightforward. Therefore, we conducted several experiments using RMSE and R 2 metrics to identify the most suitable value.

Each bidirectional LSTM layer consists of 128, 128, and 64 units, respectively. Dropout layers are employed to mitigate overfitting. In our experiments, LSTM units were randomly deactivated during training, with a dropout rate of 20%.

The weight kernel was initialized using Glorot uniform (Glorot and Bengio, 2010) to prevent the activation outputs of the layers from experiencing gradient explosion or vanishing during the forward pass.

4. Experimental results

In this section, we deploy qualitative and quantitative results to demonstrate the efficiency of the Attention-BiLSTM on SST value prediction. First, we introduce a brief description of evaluation metrics as well as the experimental parameter settings, and then we compare the performance of Attention-BiLSTM with the standard LSTM, Transformers, and other forecasting methods.

4.1. Evaluation metrics

To analyze and select the best-performing models, the most common evaluation metrics used in time series predicting are adopted. Given x the actual observation, y the predicted observation, and N the total number of testing set, the evaluation criteria are computed based on the following eqs.

A Mean Absolute Error (MAE) represents the average of absolute errors of prediction for a set of observations and predictions.

Formula not decoded

Model A	Number of BILSTM layers
Model B	1
Model C	1
Model D	3

Model A	Number of dropout layers
Model B	1
Model C	1
Model D	3

Model A	MAE	MAPE	RMSE	R 2
Model A	0.31330	0.014951	0.435388	0.989899
Model B	0.233684	0.011443	0.234729	0.998343
Model C	0.324140	0.015948	0.453385	0.987964
Model D	0.204140	0.010046	0.280905	0.995583

Model	Parameter	Value
XGBoost	max depth	3
	learning rate	0.1
	n estimators	100
RF	n estimators	100
SVR	criterion	absolute_error
	degree	linear

All experiments are conducted on the testing set, using identical experimental settings. This ensures a fair comparison between different models and methods. By using the same testing set and experimental conditions, the results can be directly compared to evaluate the performance of the Attention-BilSTM method.

To verify the performance of our proposed approach, we accomplish different scenarios:

· Model A: it represents the Attention-BilSTM model with three bidirectional LSTM layers without using dropout layers;
· Model B: it represents the Attention-BilSTM model with two bidirectional LSTM layers and one dropout layer;
· Model C: it represents the Attention-BilSTM model with one bidirectional LSTM layer and one dropout layer;
· Model D: it represents the Attention-BilSTM model with three bidirectional LSTM layers and two dropout layers.

Table 6 depicts an ablation study that involves systematically removing or disabling specific layers within our neural network to analyze their individual impact on the model's performance. The goal is to understand the significance of each layer and its contribution to the overall model performance.

In Table 7, we remark that dropout layers increase the achievement

Model	MAE	MAPE	RMSE	R 2
XGBoost	0.317217	0.017061	0.433130	0.927105
RF	0.337023	0.018147	0.450764	0.921049
SVR	0.318378	0.017125	0.425294	0.929719
LSTM	0.312283	0.016706	0.442082	0.922751
BILSTM	0.306279	0.016511	0.419128	0.930565
Attention-BilGRU	0.313244	0.016939	0.415326	0.931819
Transformers	0.325090	0.016726	0.426211	0.928198
Attention-BilSTM	0.300033

of our model by improving the generalization performance of AttentionBilSTM. Also, the use of three bidirectional LSTM layers produced a stable model that provided good results.

4.3. Results and discussion

To assess the effectiveness of the proposed Attention-BilSTM method, a benchmark comparison is conducted against six other methods. The benchmark includes two traditional machine learning models: Support Vector Regression (SVR) and Random Forest (RF). Additionally, a gradient boosting method called Extreme Gradient Boosting (XGBoost) is included. Three deep learning methods specifically designed for time series data, Long Short-Term Memory (LSTM), Attention-BiGRU (Bi-Gate Current Unit), and Transformers are also part of the benchmark.

In the initial experiments, five evaluation metrics are used to compare the performance of these methods: Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), Root Mean Squared Error (RMSE), R $^{2}$, and execution time (in seconds) for both training and inference stages.

Furthermore, Table 8 provides a summary of the best parameters used during the training of the RF, XGBoost, and SVR models. These parameters are determined through experimentation and tuning to achieve optimal performance for each method.

Tables 9, 12, 11, and 10 provide insights into the performance of different models across various regions in Morocco. It is observed that the RF model generally performs the worst, exhibiting higher error values and a lower determination coefficient. On the other hand, the Attention-BilSTM consistently achieves the highest R 2 in all Moroccan regions, indicating a good fit for the SST observations. Notably, the Karit Arkmane region stands out with an R 2 of 0.995383, indicating highly accurate sea temperature predictions.

While training deep learning models can be computationally intensive and time-consuming, it is crucial to note that once a model is trained and the results are significant, it can be saved and used for inference. In terms of inference time, as shown in Tables 9, 10, 12, and 11, Attention-BilSTM model outperforms others in terms of accuracy, despite requiring a longer training process. On average, the AttentionBilSTM model can forecast 230 days in less than 3 s, demonstrating its efficiency in making predictions in a timely manner.

Figs. 7, 10, 9, and 8 visually represent the predicted SST values for the Agadir, Casablanca, Karit Arkmane, and Tangier regions, respectively. These plots provide further evidence of the quality of the predicted SST values by the Attention-BilSTM model, as they closely align with the actual SST values.

The proposed Attention-BilSTM model has demonstrated remarkable predictive performance across the testing set of four different Moroccan regions. The use of the attention mechanism in the model has contributed to improved predictions, as it allows the model to focus on important features. Compared to other models, the Attention-BilSTM model proves to be a highly effective solution for predicting SST values. Overall, the experimental results, visualizations, and comparative analysis indicate that the proposed Attention-BilSTM method accurately predicts SST values in various Moroccan regions. The inclusion of

Model	MAE	MAPE	RMSE	R 2	Training	Inference
XGBoost	0.310400	0.017155	0.444201	0.954140	0.11	0.01
RF	0.315441	0.017460	0.454354	0.952020	14.09	0.01
SVR	0.298017	0.016496	0.424580	0.958102	1.40	0.01
LSTM	0.300582	0.016716	0.431939	0.955702	497.14	1.48
BILSTM	0.298235	0.016482	0.420803	0.957957	487.34	1.45
Attention-BIGRU	0.299526	0.016551	0.424332	0.957248	357.57	1.75
Transformers	0.338829	0.018947	0.464905	0.948682	387.79	1.47
Attention-BILSTM	0.309356	0.017203	0.422227	0.957672	453.72	2.03

Model	MAE	MAPE	RMSE	R 2	Training	Inference
XGBoost	0.213739	0.010608	0.301511	0.994718	0.24	0.02
RF	0.225040	0.011079	0.302400	0.949681	23.83	0.01
SVR	0.212706	0.010399	0.289286	0.995114	3.18	0.01
LSTM	0.225529	0.011333	0.292770	0.994984	343.88	1.42
BILSTM	0.212948	0.010609	0.284854	0.995252	392.92	1.21
Attention-BIGRU	0.206316	0.010158	0.282809	0.995320	393.35	1.83
Transformers	0.205884	0.010088	0.284455	0.995265	389.44	2.16
Attention-BILSTM	0.204140	0.010046	0.280905	0.995383	453.7	2.11

Model	MAE	MAPE	RMSE	R 2	Training	Inference
XGBoost	0.170690	0.008851	0.243260	0.988788	0.05	0.001
RF	0.176252	0.009135	0.250328	0.988126	14.29	0.01
SVR	0.165814	0.008582	0.230848	0.989263	1.44	0.01
LSTM	0.203848	0.010782	0.26311	0.986738	446.78	1.83
BILSTM	0.168753	0.008736	0.232849	0.989126	480.56	0.38
Attention-BIGRU	0.180524	0.009448	0.244791	0.988521	401.15	2.70
Transformers	0.172817	0.008909	0.244732	0.988526	342.30	1.93
Attention-BILSTM	0.168371	0.008775	0.237012	0.989289	452.57	2.77

the attention mechanism enhances the model's performance, making it a superior choice compared to other models considered in the experiments.

Nevertheless, in this work, we encounter three limitations: the lack of availability of more data, the hyperparameter setting, and climate change. Dealing with limited data in time series analysis can indeed present significant challenges. The nature of time series data, often characterized by its sequential and temporal dependencies, can make the analysis complex, especially when there are few observations.

Moreover, the hyperparameter setting is an essential part of the deep learning workflow, influencing a model's generalization, performance, and computational efficiency. It often requires a balance between the exploration of various settings and the computational resources available for experimentation. In these experiments, we used only four datasets from four Moroccan cities. However, the hyperparameter setting will be very complex when we extend our work to all seaside towns. Finally, climate change is a major and complex problem that can affect the prediction model. It refers to long-term changes in weather

patterns on a global or regional level. These changes include variations in average temperatures, removals, extreme weather events, and climate patterns.

5. Conclusion

In this paper, we propose a novel approach for sea surface temperature (SST) prediction utilizing deep learning techniques. Our method

incorporates the Bidirectional Long Short-Term Memory (BILSTM) deep recurrent neural network model along with the attention mechanism. The attention mechanism plays a crucial role in enhancing prediction accuracy by capturing relationships between historical and future SST values. Through extensive experimentation, we demonstrate that our proposed model outperforms existing SST prediction approaches. However, our method encounters some shortcomings. Indeed, our study relied only on four data sets provided by four Moroccan cities, but its

extension to all coastal cities considerably increases the complexity of tuning hyperparameters in deep learning models. Additionally, climate change emerges as a complex and influential factor impacting climate models due to changes in global or regional weather patterns. The prediction of our model is very close to reality because it is currently learned on stable data. One of the most obvious impacts of climate change is increased SST. The oceans will absorb much of the excess heat trapped by greenhouse gases, causing SST to increase over time. In this case, our model will not be able to predict future SST values well.

In future work, we plan to explore the application of the neural prophet model to Moroccan SST data as well as other marine datasets such as Pirata (the Prediction and Research Moored Array in the Atlantic). By testing the neural prophet model, we aim to further enhance our understanding and predictive capabilities in the domain of SST forecasting. This will contribute to the advancement of marine research and enable better predictions for various regions and datasets.

CRediT authorship contribution statement

Nabila Zrira: Writing -review & editing, writing draft, original methodology, Investigation, Data curation, Conceptualization. Asia Kamal-Idrisi: Writing -original draft, Validation, Methodology, Investigation, Formal analysis, Conceptualization. Rahm Farsi: Writing -review & editing, Writing -original Conceptualization.

Haris Ahmad Khan: Writing -review & editing, Writing -original

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

Data will be made available on request.

Acknowledgment

The authors would like to thank the seasonal temperature website for providing Moroccan marine environment observation data https://sea temperature.info.

References

Ahmed, Dozdar Mahdi, Hassan, Masoud Muhammed, Msatafa, Ramdani Jadham, 2022. A review on deep sequential models for forecasting series data. Appl. Comput. Intell. Soft Comput. 2022.

Ali, Ahmed, Fathalla, Ahmed, Salah, Behkim, Mahamed, Eloumodesou, Ersaya, 2021. Marine data prediction: an evaluation of machine learning, deep learning and statistical predictive models. Comput. Intell. Neuroscience of daily sea surface data.

Aparra, S.G., D'souza, Selirra, N.B., 2018. Prediction of daily sea surface temperature using artificial neural networks. Int. J. Remote Sens. 59 (12), 4241-4243.

Bahdanou, Dzmatry, Cho, Kyunghyun, Bengio, Yoshua, 2014. Neural machine translation by jointly learning to align and translate, arXiv preprint arXiv:1409.0473.

Choi, Youngin, Park, Younghin, Whaag, Jeajondong, Jijkim, Keuin, Euyhun, 2022. Improving ocean forecasting using deep learning and numerical model for sequential analysis.

Eo, M. J. Sci. 10 (4), 450. Magn. A. S. G. 19, Int. J. Eng. 40, 55.

Eo, T. Hong, Huang, Bingu, Hang, Xiang, Zhu, Junming, Chen, Wang, Zu, Huan, Zhang, Wen, 2022. Hybrid deep learning model for the bias correction of fast numerical forecast products using satellite models. Sens. 14 (6), 139.

Galasi, Andrea, Lippi, Marco, Poi, 2020. Attention in natural language processing. IEEE Trans. Neural Netw. Learn. Syst. 32 (4012-4398).

Glorot, Xavier, Benoïde, Yoshua, 2010. Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirty-Thirty Conference on Artificial Intelligence and Statistics, pp. 249-256. J.R.M Workshop

and Conference Proceedings.

Graves, Alex, Schmidhuir, Jügen, 2005. Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Netw. 18 (5-6), 602-610.

Graves, Alex, Fernandez, Santiago, Schmidhuir, Jürgen, 2005. Bidirectional lstm networks improved for photoemphase classification and recognition. In: International Conference on Artificial Neural Networks. Springer, pp. 799-804. Hochreiter, Sepp, Schmidhuir, Jürgen, 1997. Long short-term memory. Neural Comput. 9 (8), 1375-1780.

Hou, Syin, Liu, Wang, Leen, Liu, Tian, Zhou, Shuong, Jian, Guhong, Qi, Rufu, Wang, Zheng, 2022. Mino: a unified spatio-temporal model for multi-scale sea surface temperature prediction. Remote Sens. 14 (10), 2371.

Kingma, Diederik P., Ba, Jimmy, 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.4908.

Krishnamurti, T.N., Chakraborty, Aridam, Krishnamurti, Ruby, Dewar, William K, Clayson, Carol Anne, 2006. Seasonal prediction of sea surface temperature anomalies using a suite of 13 coupled atmosphere-ocean models. J. Clim. 19 (23), 6069-6088.

Kun, Xiao, Shan, Tian, Yi, Tan, Chao, Chen, 2021. Attention-based long short-term memory network temperature prediction. In: 2022 International Conference on Condition Monitoring of Machinery in Non-Stationary Operations (CNN), IEEE, pp. 278-281.

Lee, Dong, Eun, Chapman, D, 2020. Henderson, N., Chen, Chen, P, 2016. Multilevel vector adaptive prediction of sea surface temperature in the north tropical Atlantic ocean and the carlsea. J. Dyn. 7 (4), 95-106. Lee, Ming-Ch, Cheng, Jia, W, He, Yeng-Ch, Sheng, Cheng, T-long, Tsao, J-Ling, H, Chen, Xu, Ming, 2022. Application attention based biltsim and technical indicators in the design and performance analysis of stock trading strategies. Neural Comput. & 1-13.

Kien, X-ming, 2022. Application attention based biltsim and technical indicators in the design and performance analysis of stock trading strategies. Neural Comput. & 1-13.

Isis, Isidler, Arujo, Maozy, Marci, Dao, 2013. Prediction of sea surface temperature in the tropical Atlantic vector by machine machines. Comput. Stat. Data Anal. 16 (18), 0179. Luong, Minh-Thang, Pham, Henry, Manning, Christopher D, 2015. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv preprint 1590.04202.

Ma, Jia, M., Jiecheng, Feng, X., Yang, Chien, Xiao, Li, Wang, 2018. No-classification and data-driven approach for collision risk early warning in vessel encounter techniques. AIR 10 (8), 18771-18873.

Nie, Qingwing, Wang, Dingsheng, Wang, Rui, 2021. Bin-chim slt model water prediction method with attention mechanism. In: Journal of Physics: Conference Series, 2078. IOP Publishing, p. 012032.

Niau, Zhoayang, Zheng, Guoqiang, Hu, Yu, 2021. A review on the attention mechanism of deep learning. Neurocomputing 452 (0), 48-62.

Noor, Fahima, Haq, Sanra, Raikhab, Mohammed, Ahmed, Tarik, Jamelah, S, Zaman, Zahaka, Sansha, Rubat, Yusnava, Tsufan, Ahmad, Naman, Sarkar, Dewan, Ashraf, Rahman Sadaraf, Mehta, 2022. Water level forecasting using spatiotemporal attention-based long short-term memory network. Water 14 (4), 612.

Qiao, Bailyou, Zhongqiang, Wu, Ma, Ling, Zhou, Yicheng, Sun, Yunchu, 2021. Effective ensemble learning approach for stf prediction using attention based-predm.

Front. Comp. Sci. 17 (1), 1721601.

Reneha, Shaik, 2019. River water temperature modelling under climate change using support vector regression. In: Hydrology in a Changing World. Springer, pp. 171-183.

Shia, Jihao, Yu, Yang, Jinkin, Yun, Liu, Guang, Hu, Yuan, 2022. Time series surface temperature prediction based on cyclic evolutionary network model for complex sea area. Futur. Inteber 14 (3), 96.

Sit, Muhammad, Demir, 2020. A comprehensive review of deep learning applications in Demir, Ibrahim, 2020. A comprehensive review of deep learning applications in hydrology and water resources. Water Sci. Technol. 82 (12), 2635-2670.

Tule, Alexand, Elfathair, Elfathair, A.B. 2018. Seasonal precipitation forecast over Morocco. Water Resour. Res. 54 (11), 91931-9186.

Wang, Feng, Tax, David M., 2012. Survey on the attention based rnn model and its applications in computer vision. arXiv preprint arXiv:1602.08161.

Wang, Yequan, Hua, Minlie, Zhi, Xiaowu, Zhao, Li, Xiao, Zheng, 2016. I-18. I-19. I-20. I-21. I-22. I-23.

Empirical Methods in the Natural Language Processing, pp. 606-615.

Wang, Xiaowang, Lei, Wang, Zhang, Zhi, Wei, Chen, Kuo, Jin, Yingying, Yan, Yun Liu, Jing, Zwing, 2022. A extended fusion method for sea surface temperature prediction on the East China Sea. Appl. Sci. 12 (12), 5905.

Wolf, Stefan, O'Donna, Ghez, Hahal, Chen, Bei, 2020. Statistical and machine learning ensemble modeling to forecast sea surface temperature. J. Mar. Syst. 208 (0),

103473.

Xiao, Chianggai, Cheng, Nengcheng, Chui, Hu, Wang, Ke, Zewei, Xu, Cai, Ying, Lei, Xu, Chen, Zeigang, Gong, Jianya, 2019. A spatiotemporal deep learning model for sea surface temperature field prediction using time-series satellite data. Environ Model Softw. 120 (10), 04502.

2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1480-1489. Zhang, Xiaoyu, Li, Yongqing, Frey, Alejandro C., Ren, Peng, 2021. Sea surface temperature prediction with memory graph convolutional networks. IEEE Geosci. Remote Sens. Lett. 19 (0), 1-5.

Zheng, Gang, Li, Xiaofeng, Zhang, Rong-Hua, Liu, Bin, 2020. Purely satellite data-driven deep learning forecast of complicated tropical instability waves. Sci. Adv. 6 (29) eaba1482.